Test of Performance Strategies (TOPS): Instrument refinement using confirmatory factor analysis

Background and purpose: A recent confirmatory factor analysis (CFA) of the Test of Performance Strategies (TOPS) by Lane, Harwood, Terry, and Karageorghis [2004. Confirmatory factor analysis of the Test of Performance Strategies (TOPS) among adolescent athletes. Journal of Sports Sciences, 22, 803–812] provided only mixed support for structural integrity of the TOPS. The objectives of the present paper were to further examine the instrument’s structural integrity and enhance it if necessary. Method and results: In a pilot study, a sample of North American athletes completed the TOPS. Results revealed poor fits during analysis of the competition and practice subscales. In Study 1, a number of new items were developed and a new competition subscale (distractibility) introduced, to address the problems identified and create the TOPS 2. CFAs of responses from a sample of Australian, North American and British athletes provided much stronger support for the factorial validity of the TOPS 2 inventory. However, the distractibility subscale suffered from poor factor loadings and reliability, and so was removed from further analysis. In Study 2, the factorial validity of the TOPS 2 was confirmed on a new sample of Australian athletes. Conclusions: The TOPS 2 appears to be an improvement over the TOPS. Implications of the results for practitioners are discussed, and future research directions are recommended. 2009 Elsevier Ltd. All rights reserved. Psychological inventories, based on athlete self-report, are an important means of assessing the cognitive and affective states of athletes (Vealey & Garner-Holman, 1998). Indeed, some argue that psychological assessment is an essential requirement for any sport psychology intervention (e.g., Beckmann & Kellmann, 2003). Traditional forms of assessment in sport psychology were based on the assumption that personality traits or states determine an individual’s pattern of behaviour (Tkachuk, Leslie-Toogood, & Martin, 2003). Thus, sport psychologists used some instruments from clinical and counselling psychology, such as the 16 Personality Factor Questionnaire (Cattell, 1949), the State-Trait Anxiety Inventory (Spielberger, Gorsuch, & Lushene, 1970), and the Profile of Mood States (McNair, Lorr, & Droppleman,1971). Other instruments were developed specifically to measure such constructs with athletes, including the Athletic Motivation Inventory (Tutko, Lyon, & Ogilvie, 1969), the Test of Attentional and Interpersonal Style (Nideffer, 1976), and the Sport Competition Anxiety Test All rights reserved. (Martens, 1977). However, the suitability of these measures in sport psychology research and practice has been questioned because of their clinical focus, absence of athlete norms, questionable psychometric properties, and inconsistencies in findings (Ford & Summers, 1992; LeUnes & Nation, 1989; Tkachuk et al., 2003). Many sport psychologists have turned to inventories that measure sport-related behaviours rather than any underlying personality dimensions that might be linked to those behaviours. There is considerable interest in instruments targeting psychological skills and strategies in sport as they are likely to differentiate more and less successful athletes, and provide evidence regarding the efficacy of psychological skills training programs. Such instruments include the Psychological Skills Inventory for Sport (PSIS, Mahoney, Gabriel, & Perkins, 1987); the Athletic Coping Skills Inventory (ACSI-28, Smith, Schutz, Smoll, & Ptacek, 1995); the Ottawa Mental Skills Assessment Tool (OMSAT-3, Durand-Bush, Salmela, & Green-Demers, 2001); and the Test of Performance Strategies (TOPS, Thomas, Murphy, & Hardy, 1999). Such instruments must have sound psychometric properties to be of any theoretical or practical value. L. Hardy et al. / Psychology of Sport and Exercise 11 (2010) 27–35 28 The TOPS was designed to measure a comprehensive range of psychological skills and techniques, and their strategic use by athletes both in competition and at practice. Subscales were developed targeting eight of the most salient psychological skills and processes thought to underlie successful athletic performance. Interested readers are referred to Thomas et al. (1999) for more on the theoretical basis for the inclusion of the particular psychological skills within the TOPS scales. These skills are goal setting, relaxation, activation, imagery, self-talk, attentional control, emotional control, and automaticity. All of these skills and strategies are measured at practice, but as a result of exploratory factor analyses by Thomas et al., negative thinking rather than attentional control is measured in competition. Responses from the original sample of athletes revealed moderate correlations among many of the TOPS subscales – automaticity being a notable exception, as well as similarities in the patterns of use of these skills and strategies at practice and in competition (Thomas et al., 1999). Those performing at international and national standards reported more use of most psychological skills and strategies and less negative thinking than club/recreational athletes (Thomas et al., 1999). Automaticity was again an exception. Older athletes reported more automaticity in the execution of skills than younger athletes. However, it is not clear why male athletes performing at international and national standard reported lower levels of automaticity in competition than less skilled male athletes. There was no clear pattern of differences in female athletes. Subsequent research has provided further evidence of the construct validity and internal consistency of the TOPS subscales. Gould, Dieffenbach, and Moffett (2002) included the TOPS in a battery of inventories used to examine the psychological characteristics of Olympic champions. These elite athletes were high on goal setting, activation, relaxation, and emotional control in competition, and on goal setting and attentional control at practice. Compared to the norms for international athletes (Thomas et al., 1999), the Olympic champions scored substantially higher on automaticity, emotional control, and relaxation in competition, and lower on negative thinking. In addition, they scored substantially higher on goal setting and attentional control at practice. Gould et al. (2002) also interviewed the Olympic champions, as well as a coach, and a parent, sibling, or significant other for each of the athletes. The themes emerging from these interviews provided triangulation for many of the quantitative findings in this study, which in turn corroborated previous research identifying mental skills as key components of peak performance (Williams & Krane, 2001). Jackson, Thomas, Marsh, and Smethurst (2001) investigated relationships between the use of psychological skills in competition, athlete self-concept, the flow experience, and performance. They found that the more athletes use psychological skills in their sport, the more likely they are to experience flow. All of the TOPS subscales, with the surprising exception of automaticity, contributed significantly to the predictions of flow. Further, there were strong correlations across instruments for measures of similar constructs (e.g., goal setting on the TOPS was correlated with the clear goals dimension of the Flow State Scale) and no correlations between TOPS subscales and dissimilar constructs (e.g., transformation of time). Internal consistency of the TOPS competition subscales was generally higher than in the original study (alphas ranged from 0.77 to 0.88). Fletcher and Hanton (2001) used the TOPS to examine whether the use of psychological skills was related to athletes’ competitive anxiety responses. They found that athletes who made high use of relaxation, self-talk, and imagery skills differed significantly in their competitive anxiety responses from those who made low use of these psychological skills. Those who made high use of relaxation reported lower levels of cognitive and somatic anxiety, which they interpreted as facilitative rather than debilitative. They also reported higher levels of self-confidence than thosewhomade low use of relaxation. Those making high use of self-talk and imagery were more self-confident than those making low use of these skills. No differences in competitive anxiety responses were found for goalsetting usage. The TOPS responses of a small sample of male soccer players competing at the 2000 Amputee World Cup were used to examine relationships between psychological skills, self-efficacy, and performance (Lowther, Lane, & Lane, 2002). Higher use of relaxation strategies in competition was associated with higher levels of self-efficacy, and the use of imagery at training was significantly related to self-ratings of performance in competition. The importance of using mental skills at practice, not just in competition, is also apparent in other research with the TOPS. Frey, Laguna, and Ravizza (2003) demonstrated that athletes’ use of mental skills was related to their perceived success at practice and in competition. Themore athletes usedmental skills at practice, the more successful they perceived themselves to be, not only at practice, but also in competition. Frey et al. (2003) urged sport psychology consultants to make coaches aware of the relationship between mental skill use in practice and success in competition, so that athletes would use mental skills when practising their physical skills, thereby enhancing the quality of practice. The TOPS has become one of the most popular tests in sport psychology (Weinberg & Gould, 2003), with very promising subscales (Bond & Sargent, 2004). The preliminary analyses reported by Thomas et al. (1999) showed that the subscales had good construct validity, which is also evident from subsequent research. Some of the findings in relation to automaticity, however, have been contrary to expectations, suggesting that further attention should be given to the items on this subscale. Most of the subscales have demonstrated good levels of internal consistency, particularly given that they are each comprised of just four items. However, two of the subscales, measuring activation and automaticity at practice, displayed only moderate reliability in the original study (Cronbach’s alphas of 0.66 and 0.67 respectively), again suggesting that these subscales warrant further attention. Having said all this, it must be noted that the TOPS subscales were based on exploratory factor analyses. Although it is often necessary to use exploratory factor analysis as a preliminary step in inventory development, this should be followed by confirmatory factor analysis (CFA) to test the proposed factor structure (Schutz & Gessaroli, 1993). Moreover, the CFA should be conducted on data from an independent sample to cross-validate the factor structure (Schutz & Gessaroli, 1993). Thomas et al. (1999) thus signalled the need for further research using CFA to examine the factorial validity of the TOPS. Lane, Harwood, Terry, and Karageorghis (2004) recently undertook such a confirmatory approach, and examined the factorial validity of the TOPS with data from adolescent British athletes. There were mixed results regarding the hypothesised measurement models for competition and practice. The root mean square error of approximation (RMSEA) indicated good model fit for competition, and adequate fit for practice. However, incremental fit indices (the robust comparative fit index, RCFI; and the Tucker-Lewis index, TLI) indicated that both models could be significantly improved. The analyses provided strong support for the automaticity, goal setting, relaxation, and self-talk competition subscales; indicated scope for improvement in the emotional control, imagery, and negative thinking subscales; and provided no support for the fit of the activation subscale. Corresponding analyses for the practice subscales provided strong support for the attentional control, emotional control, goal setting, imagery, and L. Hardy et al. / Psychology of Sport and Exercise 11 (2010) 27–35 29 self-talk subscales; indicated scope for improvement in the automaticity and relaxation subscales; and again provided no support for the activation subscale. All competition subscales displayed adequate internal consistency (alphas ranged from 0.69 to 0.82), but two of the practice subscales, activation and automaticity, had low internal consistency (alphas of 0.57 and 0.62 respectively). Most of the items (84%) had good to excellent factor loadings and standard errors. Only 3 of the 64 items were considered poor – two on the activation in practice subscale, and one on the automaticity in practice subscale. The participants in the Lane et al. (2004) study were all younger than 18 years of age, and the authors questioned whether the language used in some items might be inappropriate for adolescents. For example, they questioned whether adolescents understand the concept of performing ‘‘on automatic pilot’’, even though the CFA provided strong support for the automaticity competition subscale. Clearly, the factor structure of the TOPS needs to be further examined using data from athletes ranging in age and ability. In addition, subtle cultural differences can affect instrumentation (Gauvin & Russell, 1993). In the present paper, we therefore briefly report a pilot studywhich used CFA to examine the factorial validity of the TOPS for North American rather than Australian athletes from whom the original data were obtained. Our first study then examined the factorial validity of a revised version of the TOPS (TOPS 2) which contained several new items, and a new subscale developed to address measurement issues identified with the initial instrument (these issues are discussed in detail in the Method section of Study 1 under Instrument Refinement). The participants were a mixed sample of American, Australian, and British athletes. Finally a second study used a sample of Australian athletes to confirm the factorial validity of the TOPS 2.